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ATTACHMENT FOR SUBSTITUTE SPECIFICATION 

(Marked-up) 



AUTOMATIC EVALUATION METHOD, AUTOMATIC 
EVALUATION SYSTEM, AND STORAGE MEDIUM STORING 
AUTOMATIC EVALUATION PROGRAM 



FIELD OF THE INVENTION 



[0001] 



The present invention relates to an automatic evaluation 



method, an automatic evaluation system, and a storage medium storing an 
automatic evaluation program, which can automatically evaluate a program 
operating on a target system from an input event such as a key input and a 
reference output resulting from this input event on, for example, an output 
screen. 



widely used in various devices such as electrical domestic appliances. For the 
purpose of operating the microcomputer in accordance with the specification of 
the device in which it is installed, such as a peripheral device or the like, an 
application program is written in a built-in ROM (Read Only Memory). Liquid 
crystal display devices (hereinafter referred to as an LCD: Liquid Crystal Display) 
are also installed in various devices such as electrical domestic appliances. 
Thus, whon When an input event such as a key input by a user is inputted, the 
microcomputer causes an output screen corresponding to this input event to be 
outputted on the LCD by the application program. Thon, i n tho caso whoro When 



BACKGROUND OF THE INVENTION 



[0002] 



In recent years, microcomputers have been installed and 
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the operation by the application program of the microcomputer is recognized, the 
microcomputer must recognize output results from a number of input events 
corresponding to the specification of the installation device. 

[0003] Thus, in the development of the microcomputer, the 
development of the application program, together with the development of the 
hardware, is also important. In the development of the application program, an in- 
circuit emulator (hereinafter referred to as an ICE: In Circuit Emulator) or the like 
is used. The ICE can emulate the operation of the application program on a 
target board. As described above, in the operation confirmation of the application 
program, there are a number of input events to be confirmed. Thus, i n tho caso 
wh e r e It is time consuming for the operator to directly input[[s]] the input events 
by using the ICE[[,]. it takos a l ong t i mo to i nput, and In addition, there is also a 
possibility that the operator may makelTslI an input mistake. 

[0004] Bosidos this, in th e caso whoro It is also time consuming for 
the operator to confirm[[s]] the input event and the emulation result through the 
screen output or the like[[,]]. it takos a long t i mo, and thoro There is also a 
possibility that the operator may make[[s]] a confirmation mistake. Further, in the 
evaluation of the application program by this operation confirmation, the 
evaluation for the same input event is repeated in order to improve the evaluation 
system. Thon, i n ordor to mako tho oporat i on conf i rmat i on of tho applicat i on 
program with high accuracy and off i ciont l y, an automat i c ova l uation systom i s 
usod i n wh i ch a numbor of input ovontc can bo ropoatodly and automat i ca ll y 
i nputtod, and output rosu l ts of tho i nput ovonts aro automat i ca ll y ova l uatod. 
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[0005] Accord i ng to th e forogo i ng automat i c eva l uat i on s y s t e m, th e 
appl i cation program oporat i ng on tho targot system can bo automatica l ly 
ova l uatod by us i ng simu l ation rosu l ts of a simu l ation un i t. Thus, it is nocossary to 
prov i do a momory which can bo common l y accossod by tho automat i c ova l uat i on 
syst e m and tho simu l at i on un i t, and by th i s, access botw ee n tho automat i c 
o va l uation systom and tho simulat i on unit bocomos pocsiblo. At th i s t i mo, th o 
simulat i on un i t carrios out norma l procoss i ng to porform a simu l at i on of tho i nput 
ovont and to output tho s i mu l at i on rosu l t. Accord i ng l y, it i s not nocossary to 
incorporat e a procoduro for automatic ovaluation i n th e app l icat i on program. 

[0006] I n tho forogoing automat i c ova l uation syctom, i t is nocossary 
that an i nput ovont f il o bo proparod i n advanco, and roforonco data 
corrospond i ng to tho input ovont f i lo bo proparod. Thon, an i nput ovont is 
soquont i ally transm i tted to tho s i mulator, rosu l t data i s rocoivod by roforr i ng to 
the display scroon (d i sp l ay momory) on which tho rosult of tho simu l at i on i s 
rof l octod, and tho rosult is comparod with tho r e ference data proparod in 
advanco so that an automatic ovaluat i on is carr i od out. 

[0007] Now, in actual data, ovon i f a koy i nput i s not actual l y mado, 
tho disp l ay scroon i s rowritton. That i s, for oxamplo, thoro is a bl i nking cursor or a 
charactor moving around on tho scroon, and th i s is an i nput ovont othor than a 
koy i nput. For oxamp l o, i n tho caso of a b l ink i ng cursor, s i nco it i s rowritton into 
two k i nds of scroon content s of a non - r e verso pattorn and a rovor se patt e rn by 
an app li cation program oporating on a targot program, i n tho caso whoro tho 
automatic ova l uation systom roads th i s at a suitab l o t i m i ng, an accurato 



3 



Attorney Docket No. 9319S-000231 

automat i c eva l uat i on cannot bo carriod out. 

[0008] Accord i ng l y, i n ordor to obta i n a h i gh l y ro l iab l o ova l uat i on, i t 
is nocossary to road display scroon data at a t i m i ng whon tho result of tho 
s i mulat i on b e com e s dof i n i to and tho rowr i t i ng of a scroon is comp l otod. Thus, 
conv e nt i ona ll y, data of a port i on corr e spond i ng to tho b l ink i ng cursor was 
subjoctod to a mask procossing or tho li ko so that i t d i d not bocomo an objoct of 
e valuat i on. A l tornat i vo l y, such a mothod was U 6e d so that a program op e rat i ng 
on a target systom was modifiod to stop tho b l ink i ng. 

SUMMARY OF INVENTION 

[0009] Thus, i n tho formor, tho accuracy of tho ova l uation systom i s 
l oworod, and in tho l attor, tho qua l ity of tho program is dogradod. 

[0010] The present invention has been made in view of the above 
circumstances and has an object to provide an automatic evaluation method, an 
automatic evaluation system, and a storage medium storing an automatic 
evaluation program, in which evaluation accuracy is improved by setting, for each 
input event, the number of states of an output screen which the input event can 
take (kinds of the output screen on which the input event is reflected and which is 
renewed) as number-of-repetitions data, and repeating an evaluation for one 
input event that number of times. 

DISCLOSURE OF INVENTION 
[0011] In order to solve the above problems, an automatic 
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evaluation method is an automat i c ovaluat i on mothod provided for automatically 
evaluating a program operating on a target system by referring to an output 
scr ee n as a result of a simulation on an output screen, which correspond[[ing]]s 
to an arbitrary input event[[,]]. i n wh i ch tho s i mulation is porformod, roforonco 
Reference to the output screen is made [[by]] a number of times corresponding to 
the number of valid states of the output screen on which the input event is 
reflected and which is renewed[[,]]. and tho The reference results [[is]] are 
successively compared with reference data corresponding to the number of times 
valid states of the output screen, which is prepared in advance, so that In this 
way, an automatic evaluation is carried out. 

[0012] According to this automatic evaluation method, even if the 
state of the output screen [[is]] change[[d]]s for one input event, mask processing 
is not performed[[,]]. and i t a l so bocomos It is unnecessary to modify the 
program operating on the target system^,]], so that Thus, the burden placed on 
a programmer is reduced. 

[0013] Further, the evaluation is carried out one or more times for 
one input event, and eventually, evaluation accuracy can be improved. 

[0014] Further, according to an automatic evaluation method, the 
number of times is set stored together with data of the input event data . 

[001 5] Accord i ng to this automatic ova l uat i on mothod, for For every 
input event, wh e th e r a stat e whoro i n tho output scro e n ronow e d e x i sts i s sot as 
numbor - of ropot i t i ons data, and tho ova l uat i on can bo ropoatod for ono i nput 
ovont by tho numbor of timos the number of valid states of the output screen 
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expected to result from the input event is stored together with the input data. The 
evaluation can be repeated for one input event by the number of valid states of 
the output screen expected to result from the input event , so that the evaluation 
accuracy can be improved. 

[0016] I n ordor to so l vo tho abovo problom, [[a]]An automatic 
evaluation system is an automat i c e va l uat i on sy s t e m provided for automatically 
evaluating a program operating on a target system by referring to an output 
s cr ee n as a result of a simulation on an output screen which correspond[[ing]]s to 
an arbitrary input event[[,]]. wh i ch The system comprises output screen reference 
means for[[,]] whi l e tho si mu l at i on i s b e ing p e rformod, referring to the output 
screen [[by]] while the simulation is being performed, the The output screen is 
referred to a number of times which correspond[[ing]]s to the number of valid 
states [[of]] the output screen on which the input event is reflected may take, afid 
wh i ch is r e n e w e d, and The system further comprises evaluation means for 
successively comparing the reference result with reference data corresponding to 
the number of times which is prepared in advance so that the automatic 
evaluation is carried out. 

[0017] According to this automatic evaluation system, the output 
screen reference means refers to the output screen by the number of times 
corresponding to the number of the output screen states which one input event 
can take, and the evaluation means repeats the evaluation by the number of 
times corresponding to the states. Thus, even if the state of the output screen is 
changed for one input event, mask processing is not performed[[,]]. and i t a l so 
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bocomo s It is unnecessary to modify the program operating on the target 
system[[,]]. so that Thus, the burden placed on the programmer is reduced. 
Further, the evaluation is carried out one or more times for one input event, so 
that evaluation accuracy can be improved and by this, an automatic evaluation 
system having a high performance can be obtained. 

[0018] Further, according to an automatic evaluation system, the 
system as sot forth i n claim 3 further comprises a simulation unit which performs 
the simulation and reports a display rewriting completion event every time the 
output screen is renewed by th e numbor of t i mos . 

[0019] According to this automatic evaluation system, the output 
screen reference means repeats a reference to the output screen every time it 
receives notification of the display rewriting completion event from the simulation 
device, and transmits the reference data to the evaluation means. Thus, the 
evaluation means repeats the evaluation by the number of times corresponding 
to the states, and therefore, the evaluation accuracy can be improved. 

[0020] I n ordor to so l vo tho prob l om, a A storage medium storing 
an automatic evaluation program is a storago modium storing an automatic 
ova l uat i on program provided for automatically evaluating a program operating on 
a target system by referring to an output screen as a result of a simulation 
corresponding to an arbitrary input event[[,]]. and tho The automatic evaluation 
program comprises a step of reading an input event and reference data prepared 
in advance for the input event, a step of successively transmitting the read input 
event to cause execution of the simulation, a step of performing the simulation 
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and referring to the output screen by the number of times corresponding to the 
number of states of the output screen on which the input event is reflected and 
which is renewed, and a step of carrying out an automatic evaluation by 
successively comparing the reference result with reference data corresponding to 
the number of times which is prepared in advance. 

[0021] According to the storage medium storing the automatic 
evaluation program, the automatic evaluation can be carried out by using as an 
input event the simulation result of a simulator for th e i nput e v e nt . Further, even if 
the state of the output screen is changed for one input event, mask processing is 
not performed[[,]]. and i t It is a l so b e com e s unnecessary to modify the program 
operating on the target system[[,]]. so that Thus, the burden placed on the 
programmer is reduced. Besides this, the evaluation is carried out one or more 
times for one input event, so that evaluation accuracy can be improved. 

[0022] Further, the automatic evaluation program further comprises 
a step of referring to the output screen every time a display rewriting completion 
event is received from a simulator, and repeating the automatic evaluation. 

[0023] According to the storage medium storing the automatic 
evaluation program, the automatic evaluation system receives the display 
rewriting completion event, so that it can capture the result data of the simulation 
only at a t i ming time when the simulation result becomes definite, in any state. 
Accordingly, stable reference data can be obtained, and an evaluation having a 
high reliability becomes possible by this. 
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BRIEF DESCRIPTION OF DRAWINGS 
[0024] FIG. 1 shows the structure of a personal computer in which 
an automatic evaluation system of an e mbod i m e nt is and a simulator are 
operated. 

[0025] FIG. 2 is a view for explaining tho way of access between 
the automatic evaluation system of th e e mbodim e nt and [[a]] the simulator. 

[0026] FIG. 3 is a view showing an example of an input event and a 
reference output result, in which (a) shows a key input is shown , (b) shows a 
reference output screen of an LCD before the key input in viow (a), is shown; and 
(c) shows tho reference output screen of the LCD to tho after key input of viow 
{a} is shown . 

[0027] FIG. 4 is a function development view of a persona l 
computer in an automatic evaluation s yst e m for r e al i z i ng an automatic e valuation 
mothod accord i ng to an ombodimont unit and a simulation unit in a personal 
computer . 

[0028] FIG. 5 is a flowchart of the automatic evaluation method 
according to the embodiment. 

BEST MODES FOR CARRYING OUT THE INVENTION 
[0029] Hereinafter, embodiments of an automatic evaluation 
method of the present invention, an automatic evaluation system, and a storage 
medium storing an automatic evaluation program will be described with reference 
to the drawings. FIG. 1 is a structural view of a personal computer in which an 
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automatic evaluation system and a simulator are operated, FIG. 2 is a view for 
explaining tho form of access between the automatic evaluation system and the 
simulator, and FIG. 3 is a view showing an example of an input event and a 
reference output result, in which (a) shows a key input, is shown; (b) shows a 
reference output screen of an LCD before the key input of v i ow (a) is shown , and 
(c) shows tho reference output screen of the LCD to tho after key input of viow 
{a) is shown . 

[0030] According to the automatic evaluation system of the present 
invention and the automatic evaluation method, an application program operating 
on a target system can be automatically evaluated by using a simulation result 
from a simulator. At this t i mo, tho The simulator carries out normal processing to 
perform a simulation corresponding to an input event and to output the result of 
the simulation. Thus, it is not necessary to incorporate a function for automatic 
evaluation into the application program. Further, in this automatic evaluation 
system and the automatic evaluation method, reference to an output screen is 
made by th e a number of times corresponding to the number of output screen 
states which one input event can take[[,]]. and tho The automatic evaluation is 
repeated by the number of times corresponding to the states. Bos i dos th i s, in In 
the storage medium storing the automatic evaluation program according to the 
present invention, the automatic evaluation program is loaded into an electronic 
computer from this the storage medium and [[is]] executed[[,]] so that the 
automatic evaluation system of the present invention can bo is formed, and the 
automatic evaluation by the automatic evaluation method of the present invention 
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can bo is realized. 

[0031] I nc i donta l ly, tho The target system is, for example, a 
microcomputer operating on the basis of the application program. The input 
event s are determined based on is sot i n corrocpondonco to the specification of a 
device in which the target system is installed[[,]]. and th o The target input events 
differ[[s]] depending on the input means of the installation device. The target 
input event [[is]], for example, may be a key input, a voice input or the like. The 
reference output result is a normal output of the target system in response to the 
input event, and i s sot i n corrocpondonco according to the specification of the 
device in which the target system is installed[[,]]. and tho The target reference 
output result differs depending on the output means of the installation device. 
The objective reference output result is a screen output, a voice output, or the 
like. 

[0032] I n this ombod i mont, tho The automatic evaluation system 
[[is]] may be formed as an automat i c ovaluation syctom in which by loading an 
automatic evaluation program is l oadod into a personal computer through from a 
storage medium storing the automatic evaluation program[[,]]. and tho In such 
case, the operations of the automatic evaluation program are executed in the 
personal computer to carry out an automatic evaluation. Further, in order to 
capture an input event and a reference output result, the automatic evaluation 
system uses a disk unit connected to the personal computer. Boc i dos th i s, in th i c 
ombodimont, tho The simulator i s constructod as a simu l at i on un i t i n wh i ch may 
be formed by loading a simulation program is loadod into the personal computer 
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from a storage medium storing the simulation program[[,]]. and tho The 
operations of the simulation program are executed by the personal computer to 
perform a simulation. The automatic evaluation system and the simulator are 
formed in the same personal computer. Boc i doo thic, i n ]n this embodiment, the 
target system is a microcomputer. Further, in this embodiment, the 
microcomputer is installed in a device in which an external key (button) input is 
enabled and a screen output is made onto an LCD of a game, a watch, or a data 
bank. 

[0033] First, the whole structure of an automatic evaluation system 
1 and a simulator 3 will be described with reference to FIG. 1. In the automatic 
evaluation system 1, an automatic evaluation program is read from a storage 
medium storing the automatic evaluation program by a disk unit DU of a personal 
computer PC[[,]]. and furthor, tho The automatic evaluation program is loaded 
into a main storage unit, and [[is]] executed by a central processing unit [[CP]] 
CPU (includ i ng tho ma i n storago unit) to carry out an automatic evaluation. 
Incidentally, the disk unit DU is a unit which can read and write according to the 
type of storage medium such as a floppy disk or an optical disk. The automatic 
evaluation system 1 reads an input event stored in an input event file IF from the 
disk unit DU, and transmits this input event to the simulator 3. Further, the 
automatic evaluation system 1 reads a reference output result stored in a 
reference output file OF from the disk unit DU, and compares it with a simulation 
result of the simulator 3 to the input event to carry out an automatic evaluation. 

[0034] The automatic evaluation system 1 stores an evaluation 
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result as a result log file in the disk unit DU or makes a screen output onto a 
display DP. 

[0035] In the simulator 3, a simulation program is read from a 
storage medium storing the simulation program by the disk unit DU of the 
personal computer PC[[,]]. and furth e r, th e The simulation program is loaded into 
the main storage unit and is executed [[in]] by the central procoss i ng unit CP 
CPU to perform a simulation. Bos i dos th i s, th o The simulator 3 reads an 
application program AP of a microcomputer from the disk unit DU, and simulates 
[[an lloperation by-this of the application program AP. 

[0036] When the input event is transmitted from the automatic 
evaluation system 1, the simulator 3 simulates the operation [[by]] of the 
application program AP on the basis of this input event. Thon, tho The simulator 3 
stores the simulation result in a RAM (Random Access Memory) 10 which is 
assigned to a part of the main storage unit of the personal computer PC as a 
shared memory accessed by the automatic evaluation system 1 of the present 
invention and the simulator 3 (see FIG. 2). 

[0037] In the simulator 3, an external operation is enabled by a 
keyboard KB corresponding to the key input of the installation device, and the 
LCD screen is outputted onto the display DP corresponding to the screen output 
of the LCD of the installation device. 

[0038] Incidontally, horo, s i nco Since the application program AP is 
debugged while the simulation is performed, the simulator 3 is connected to a 
debugger 2. In the debugger 2, a debug program is read from a storage medium 
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storing the debugging program by the disk unit DU of the personal computer PC, 
[[is]] loaded into the main storage unit, and [[is]] executed by the central 
process i ng unit CP CPU to perform debugging. The debugger 2 can start/stop 
the simulator 3, refer to data or rewrite data on the simulator 3, and the like. 
Further, the debugger 3 can execute the application program AP step-by-step, 
and [[can]] perform breaks. 

[0039] Here, the input event file IF and the reference output file OF 
will be described. The input event file IF and the reference output file OF are 
prepared by using an input event data preparation function and a reference data 
preparation function of the automatic evaluation system 1 , or are prepared by an 
editor in advance. 

[0040] A description will be given of a case where the input event 
file IF is prepared by the automatic evaluation system 1. First, the respective 
keys of the installation device of the microcomputer are assigned to the 
respective keys of the keyboard KB. The user prepares a number of input events 
corresponding to the specification of the installation device, and inputs the keys 
one by one. In this way, the automatic evaluation system 1 determines the kind 
of key and the input sequence as input event data for each input event. Further, 
i n tho caso whoro when the output screen of the LCD of the installation device of 
the microcomputer is changed in response to one input event, the number of 
t i mos corrospond i ng to tho numbor of valid states of the output screen to bo 
ronowod is used stored as input event data as well . Finally, the automatic 
evaluation system 1 stores the input event data for all input events in the input 
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event file IF. 

[0041] The input event file IF is given an arbitrary filename^ and i s 
stored in the storage medium, and [[is]] set in a state where it can be read from 
the disk unit DU. The input event file IF can be changed [[in]] to correspondefiGe 
to changes in the specifications of the microcomputer, changes in the 
specifications of the installation device, changes in evaluation contents, and the 
like. For example, as shown in FIG. 3(a), it is assumed that a key operation is 
performed in the order of pressing the fA} l W key, the \B] l W key and the {G} 'X? 
key. In th i s caso, i n an input event 20, as input event data, the type of the A, B 
and C keys and the input order of the keys are stored as the input event data in 
the input event file IF. Further, in the case of a blinking cursor, since there are 
two states on the output screen for this input event, the number of timos 
corrospond i ng to tho numbor of tho valid states of the output screen is stored in 
the input event file IF. 

[0042] Next, a description will be given of a case where the 
reference output file OF is prepared by the automatic evaluation system 1 . Since 
reference data in which one or a plurality of reference output results are made to 
correspond to one input event is stored in the reference output file OF, it is 
prepared in correspondence to the preparation of the input event file IF. Horo, in 
tho caso whoro Where there are a plurality of states of the output screen on 
which the input event is reflected and which is renewed for each input event, the 
reference output results are respectively set in accordance with the plurality of 
states of the output screen. For example, in the case of the blinking cursor, two 
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reference output results are set. 

[0043] Every time the user inputs an event including a key input, the 
automatic evaluation system 1 transmits, for example, the key input as the input 
event to the simulator 3. Then, the simulator 3 performs a simulation to this key 
input, and displays a simulation result on the display DP. After the display, the 
user confirms the display content of the display DP, and if correct, it becomes the 
definite reference output result. In the case where there are a plurality of 
simulation results for one input event, the plurality of simulation results are 
respectively determined as the reference output results. According to this 
embodiment, since the output means of the installation device is the LCD, the 
reference output result (reference data) is image data for the display of the LCD 
and position data on the display of the LCD. 

[0044] In the case where the application program AP is in the 
middle of development, the reference output file OF is prepared to upgrad i ng for 
upgrade the application program AP. At this time, it is assumed that in the 
reference output file OF, a bug correction portion of the application program AP, 
a specification change portion and the like are added to the automatic evaluation 
items, and the automatic evaluation containing the changed portions of the 
application program AP can be carried out. Alternatively, after the user inputs the 
key of one input event, the image data for the display of the LCD may be 
prepared as the reference output result by a reference data preparation editor of 
the automatic evaluation system 1 . 

[0045] Finally, the automatic evaluation system 1 stores the 



16 



Attorney Docket No. 9319S-000231 

reference data of all reference output results in the reference output file OF. The 
reference output file OF is given an arbitrary filename and is stored in the storage 
medium, and is set [[as]] in a state where it can be read out form the disk unit 
DU. The filename of the reference output file OF is described in the input event 
file IF, and it is read out in accordance with the input event file IF. Accordingly, 
the reference output file OF is changed in accordance with the input event file IF. 
For example, as shown in FIG. 3(b), it is assumed that before the input event 20 
is inputted, fcJUl is displayed at the upper left end on a reference output screen 
21. Then, as the input event 20, when the key input of view (a) is performed, as 
shown in view (c), [ABG_] "ABC = " is displayed as a reference output result 23 
from the upper left end to the right on a reference output screen 22 of the LCD. 
Incidentally, in the case where y ^ is the blinking cursor, [ABCJ "ABC " as a 
non-reverse pattern and [ABC ] "ABC" as a reverse pattern are displayed as the 
reference output results 23. In this case, with respect to the reference output 
result 23, the image data for the display of the LCD of [ABCJ "ABC " and [ABC ] , 
"ABC" and the display position data on the LCD screen are stored as the 
reference data in the reference output file OF. 

[0046] Next, with reference to FIG. 2, the operation of the automatic 
evaluation system 1 and the simulator 3 when the automatic evaluation is carried 
out, will be described. 

[0047] When the automatic evaluation system 1 is started by the 
user, the input event data ID stored in the input event file IF is loaded from the 
disk unit DU into the personal computer PC. Incidentally, the input event file IF is 
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specified by means of a filename supplied by the user. When the input event file 
IF is loaded, the automatic evaluation system 1 loads the reference data RD 
stored in the reference output file OF of the filename described in the input event 
file IF into the personal computer PC. 

[0048] Then, the automatic evaluation system 1 transmits one input 
event from the input event data ID to the simulator 3. API (Application 
Programming Interface) commands of the OS (Operating System) of the 
personal computer are used for the input event transmission. For example, in the 
case where the OS is Windows 98, the API command FindWindow is used to 
obtain a window handle of the simulator 3. 

[0049] Then, the API command PostMessage is used, and one 
input event in the input event data ID is transmitted to the window handle. That 
is, the transmission of the input event is enabled between the automatic 
evaluation system 1 and the simulator 3 by the API commands. Since the 
automatic evaluation system 1 and the simulator 3 use functions provided in the 
OS, such as the API commands, it is not necessary to particularly add a function 
to transmit the input event. 

[0050] Every time the input event is transmitted, the simulator 3 
simulates the operation by the application program AP on the basis of this input 
event. Then, in order to carry out a display on the display DP, the simulator 3 
temporarily stores the image data and the position data for the display of the LCD 
as the simulation result in a RAM 10. I n tho cas e whoro tho numbor of timos 
corresponding to tho numbor of When there are multiple valid states of the output 
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screen i s cot to a plura l numbor in tho i nput ovont data I D in response to the 
input event data , simulation results corresponding to the number of t i m e s valid 
states are temporarily stored. The simulator 3 also displays the image data for 
the display of the LCD stored in the RAM 10 on the display DP. The processing 
of the simulator 3 performed here is normally the same as the processing of the 
simulation of the operation by the application program AP, and a special 
processing is not performed for carrying out the automatic evaluation. 
Accordingly, a s th e app li cat i on program AP, the same on e as an app li cat i on 
program application program AP actually installed in the microcomputer can be 
used. 

[0051] I nc i dental l y, tho The RAM 10 is the main storage unit of the 
personal computer PC, and i s formod by tho RAM which can be shared by the 
automatic evaluation system 1 and the simulator 3. Accordingly, the RAM 10 can 
be accessed by the automatic evaluation system 1 and the simulator 3. That i s ]n 
this way , an exchange of simulation results is enabled through the RAM 10 
between the automatic evaluation system 1 and the simulator 3. Since the 
automatic evaluation system 1 and the simulator 3 use the RAM 10 of the 
personal computer PC, it is not necessary to add a special function which refers 
to the simulation results. The RAM 10 may be a VRAM (Video RAM) included in 
the personal computer. 

[0052] After the simulation, the automatic evaluation system 1 
refers to the simulation result stored in the RAM 10. Then, the automatic 
evaluation system 1 compares the simulation result with the reference data (the 
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image data and the position data for the display of the LCD) as the reference 
output result corresponding to the input event transmitted to the simulator 3 in the 
loaded reference data RD. The automatic evaluation system 1 determines 
whether both the results agree with each other, and evaluates the operation of 
the application program AP in response to the input event. Further, the automatic 
evaluation system 1 stores this determination result in the result log file. All the 
determination results may be stored in the result log file, or the determination 
results may be stored only in the case where the simulation results and the 
reference output results are different from each other. 

[0053] Incidentally, the automatic evaluation system 1 may display 
the simulation result and the reference output result side by side on the display 
DP in a state that the user can check. Also, the automatic evaluation system 1 
can display the determination result on the display DP. 

[0054] Every time the evaluation of one input event is ended, the 
automatic evaluation system 1 repeats the foregoing processing for the next 
input event stored in the input event data ID and carries out the automatic 
evaluation. Then, the evaluation for all input events of the input event data ID 
ends, the automatic evaluation system 1 stores the result log file in the storage 
medium such as a hard disk in accordance with the instructions of the user, and 
ends the automatic evaluation. 

[0055] A detailed description will be given of the automatic 
evaluation in the case where there are a plurality of states of an output screen for 
one input event in the automatic evaluation system 1. FIG. 4 is a function 
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development view of an automatic evaluation unit a personal computer as an 
automat i c ovaluat i on sy s tom for realizing an automatic evaluation method, and 
FIG. 5 is a flowchart showing the automatic evaluation method. In the drawing, 
blocks which are the same as those of FIG. 1 use the same numerals. 

[0056] In FIG. 4, the automatic evaluation system of the present 
invention is roughly divided into an automatic evaluation unit 1 1 and a simulation 
unit 30. The simulation unit 30 includes a built-in simulator body 31 , and the 
simulator body 31 simulates the operation of a program ( an application program 
Ap [[ ) ]] operating on a target system[[,]]. and as doscr i bod l ator, it The simulator 
body monitors a timing when data renewal of an output screen on which the 
result is reflected becomes definite, and supplies [[it]] the timing to the automatic 
evaluation unit 1 1 through a display rewriting event notification portion 32. 

[0057] The automatic evaluation unit 1 1 is formed by an evaluation 
system GGfe body 1 1 1 and an output screen reference portion 1 12. As described 
later, every time the output screen reference portion 112 obtains timing 
information (display rewriting completion event) when data renewal of the output 
screen becomes definite from the simulation unit 30, it refers to the output screen 
at tho t i m i ng , and supplies the reference data to the evaluation system cefe body 
111. The evaluation system GGfe body 1 1 1 comparos repeats comparisons of the 
reference result (simulation result) with reference data prepared in advance by 
the number of times corresponding to the number of valid states of the screen 
output on which the input event is reflected, and which i s ronowod, so that tho In 
this way an automatic evaluation is executed. 
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[0058] The simulation unit 30 is formed by the simulator body 31 for 
performing a simulation and the display rewriting completion event notification 
portion 32 for checking for a display rewriting completion and for generating and 
outputting the display rewriting completion event. Incidentally, reference numeral 
113 designates an input event file; 1 14, a reference file; and 1 15, a log file. 

[0059] Hereinafter, the automatic evaluation method will be 
described in detail with reference to the flowchart shown in FIG. 5. 

[0060] The automatic evaluation unit 1 1 reads the input event data 
ID from the prepared input event file 113 from the disk unit DU (step S51). 
Incidentally, as described above, the data [[of]] includes the number of times (n) 
corresponding to the number of states of the output screen to be renewed for one 
input event is a l so se t in the input ovont data I D . 

[0061] Next, the automatic evaluation unit 1 1 transmits the captured 
input event to the simulator 31 included in the simulation unit 30 by API 
commands (step S52). Then, the simulator 31 responds to the input event (step 
S53), and starts the simulation. 

[0062] Next, the simulator 31 generates display data (result data) 
from the simulation and renews the content of the RAM 10 (step S54). Then, the 
simulation unit 30 checks whether or not writing of the simulation result data is 
completed (step S55). Here, the simulation unit 30 continues checking until the 
writing is completed. Incidentally, the simulation unit 30 determines that rewriting 
of the simulation result data is completed when writing into the RAM 10 from the 
application program AP is not done for a certain amount of time or a cycle time. 
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Further, after the completion of rewriting of the simulation result data (display 
data), the simulation unit 30 transmits the display rewriting completion event from 
the display rewriting event notification portion 32, and stops the simulation (step 
S56). Subsequently, after the reception of the display rewriting completion event, 
the automatic evaluation unit 11 captures the simulation result data (step S57). 
After the completion of capture, the automatic evaluation unit 1 1 transmits the 
screen data reading completion notification to the simulation unit 30. Then, after 
the reception of the screen data reading completion notification, the simulation 
unit 30 reopens the simulation (step S58). 

[0063] Next, the automatic evaluation unit 11 counts checks the 
state number n (step S59). Then, in the case where the state number n is other 
than 0, the automatic evaluation unit 1 1 subtracts one from the state number n 
and continues to capture the simulation result data, and tho simulator 31 
proceeds to execute[[s]] the processing of step S54. On the other hand, in the 
case where the state number n is 0, the automatic evaluation unit 1 1 reads the 
reference data RD from the reference file 114 (step S60). Subsequently, the 
evaluation system sore body 1 1 1 of the automatic evaluation unit 1 1 compares 
the reference data RD with the simulation result data (step S61). The evaluation 
system GGfe body 111 determines whether or not the simulation result data 
agrees with the reference data RD, and evaluates the operation of the application 
program AP in response to the input event (step S62). In this evaluation, since n 
(the state number) pieces of simulation result data are captured, the evaluation 
system core 111 compares the n pieces of simulation result data with the 
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reference data RD respectively in order to perform the determination. As a result, 
in the case where all the n pieces of simulation result data agree with the 
reference data RD, the automatic evaluation unit 1 1 ends the processing, and in 
the case where they do not agree with each other, the automatic evaluation unit 
stores an error log in the log file 115 and ends the processing (step S63). 

[0064] The display rewriting event notification portion 32 included in 
the simulation unit 30 monitors a writing cycle of the result data at the same time 
as the generation of the result data[[,]]. and thus Thus , a time count is made for 
a predetermined time i n advanco to check the determination of the simulation 
result data and issue[[s]] the display rewriting completion event (step S56). The 
output screen reference portion 1 12 of the automatic evaluation unit 1 1 waits for 
the arrival of the event from the display rewriting comp l ot i on event notification 
portion 32, and captures the simulation result data at that time (step S57). 
Further, the automatic evaluation unit 1 1 counts the state number n (step S59), 
and in the case where the count value is other than 0, it continues to capture the 
simulation result data[[,]]. and whon When the count value becomes 0, it reads 
the reference data RD corresponding to the simulation result data into the 
evaluation system Gere body 111 (step S60). The evaluation system eere body 
111 compares the simulation result data with the reference data RD and 
evaluates (step S61, S62). In this evaluation, the automatic evaluation unit 11 
determines whether or not the n pieces of simulation result data agree with the 
reference data RD, evaluates the operation of the application program AP to the 
input event (step S62), and stores the result of the determination in the log file 
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115 (step S63). 

[0065] As described above, according to this automatic evaluation 
system 1, the access between the automatic evaluation system 1 and the 
simulator 3 is enabled by accessing the API commands or the RAM 10. Thus, it 
is not necessary to add a specific function to the automatic evaluation system 1 
and the simulator [[3]] 2 in order to transmit the input event from the automatic 
evaluation system 1 to the simulator [[3]] 2 and to refer to the simulation result 
data of the simulator [[3]] 2 by the automatic evaluation system 1 . Further, it is 
not necessary to install a function for automatic evaluation into the application 
program AP[[,]]. and tho The same ono as an application program actually 
installed in a microcomputer may be used. Further, according to this automatic 
evaluation system 1 , even if there are a plurality of states of an output screen for 
one input event, an automatic evaluation to the respective output screen can be 
carried out. 

[0066] The present invention is not limited to the foregoing actual 
mode, but can be carried out in various modes. 

[0067] For example, although the API commands and the RAM are 
used for the communication between the automatic evaluation system 1 and the 
simulator 3, the invention is not limited to these means but may use other means. 

[0068] Besides this, although the automatic evaluation system 1 
and the simulator 3 are formed in the same personal computer, they may be 
formed in another electronic computer such as a workstation. 

[0069] Besides this, a structure may also be adopted in which an 
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automatic evaluation program stored in another computer is downloaded in a 
personal computer forming an automatic evaluation system and a simulator 
through a network and is executed. 

[0070] According to an automatic evaluation method of the present 
invention, a simulation is performed, a reference to an output screen is made by 
the a number of times corresponding to the number of states of an output screen 
on which an input event is reflected and which is renewed, and the reference 
result is successively compared with reference data prepared in advance and 
corresponding to the number of times so that an automatic evaluation is carried 
out[[,]]. and thoroforo Therefore , even if the state of the output screen is changed 
for one input event, a mask processing is not performed[[,]]. and it ]t also 
becomes unnecessary to modify a program operating on a target system. Thus, 
the work load on a programmer is reduced, and further, since the evaluation is 
carried out one or more times for one input event, evaluation accuracy can be 
improved. 

[0071] Further, according to the automatic evaluation method of the 
present invention, for each input event, the number of kinds of states for 
renewing the output screen that exist is set as number-of-repetitions data, so 
that the evaluation can be repeated that number of times for one input event, and 
therefore, the evaluation accuracy can be improved. 

[0072] According to the automatic evaluation system of the present 
invention, the output screen reference means refers to the output screen by the 
number of times corresponding to the number of the states which one input event 
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can have, and the evaluation means repeats the evaluation by the number of 
times corresponding to the states. Thus, even if the state of the output screen is 
changed for one input event, a mask processing is not performed, and it also 
becomes unnecessary to modify the program operating on the target system, so 
that the burden placed on the programmer is reduced. Further, the evaluation is 
carried out one or more times for one input event, so that evaluation accuracy 
can be improved and by this, the automatic evaluation system having a high 
performance can be constructed. 

[0073] According to the automatic evaluation system of the present 
invention, by the display rewriting completion event, the result data of the 
simulation can be captured only at the timing when the simulation result becomes 
definite, in any state. Accordingly, stable reference data can be obtained, and a 
highly reliable evaluation is enabled by this. 

[0074] According to the storage medium storing the automatic 
evaluation program of the present invention, the automatic evaluation can be 
carried out by using the simulation result of the simulator to an input event, and 
even if the state of the output screen is changed for one input event, a mask 
processing is not performed, and it also becomes unnecessary to modify the 
program operating on the target system, so that the burden placed on the 
programmer is reduced. Besides this, the evaluation is carried out one or more 
times for one input event, so that the evaluation accuracy can be improved. 

[0075] According to the storage medium storing the automatic 
evaluation program of the present invention, the display rewriting completion 
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event is received, so that the result data of the simulation can be captured only at 
the timing when the simulation result becomes definite, in any state. Accordingly, 
stable reference data can be obtained, and highly reliable evaluation is enabled 
by this. The entire disclosure of Japanese Patent Application No. 2000-178342 
filed June 14, 2000 is incorporated herein by reference. 
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